Highly-Scalable Wait-Free Buffering Scheme for Multi-Core System Tracing

Diagnostic of complex problems involving the interaction between several applications and the operating systems in a distributed multi-core system with tracing requires to extract system events without disturbing its execution. Recording an event must therefore take negligible time (e.g. less than the duration of simple system call) and should not change the ordering of events (non locking). As a result, a highly-scalable, low-overhead, lock-free trace buffering scheme was designed and proposed for the LTTng tracer. It uses local compare-and-exchange operations for synchronization and resorts to RCU (Read-Copy Update) data structures for atomically updating control data. The cost associated with tracer execution has been benchmarked for different types of execution loads, on systems scaling up to 8 cores. The performance of the new trace buffering system has been measured and compared to other tracing systems such as DTrace, SystemTap and K42.